<!DOCTYPE html>

<html>
  <head>
    <meta charset="utf-8">
    
    <title>Testing Guidelines &mdash; NumPy v1.18 Manual</title>
    
    <link rel="stylesheet" type="text/css" href="../_static/css/spc-bootstrap.css">
    <link rel="stylesheet" type="text/css" href="../_static/css/spc-extend.css">
    <link rel="stylesheet" href="../_static/scipy.css" type="text/css" >
    <link rel="stylesheet" href="../_static/pygments.css" type="text/css" >
    <link rel="stylesheet" href="../_static/graphviz.css" type="text/css" >
    
    <script type="text/javascript">
      var DOCUMENTATION_OPTIONS = {
        URL_ROOT:    '../',
        VERSION:     '1.18.1',
        COLLAPSE_INDEX: false,
        FILE_SUFFIX: '.html',
        HAS_SOURCE:  false
      };
    </script>
    <script type="text/javascript" src="../_static/jquery.js"></script>
    <script type="text/javascript" src="../_static/underscore.js"></script>
    <script type="text/javascript" src="../_static/doctools.js"></script>
    <script type="text/javascript" src="../_static/language_data.js"></script>
    <script type="text/javascript" src="../_static/js/copybutton.js"></script>
    <link rel="author" title="About these documents" href="../about.html" >
    <link rel="index" title="Index" href="../genindex.html" >
    <link rel="search" title="Search" href="../search.html" >
    <link rel="top" title="NumPy v1.18 Manual" href="../index.html" >
    <link rel="up" title="Test Support (numpy.testing)" href="routines.testing.html" >
    <link rel="next" title="Window functions" href="routines.window.html" >
    <link rel="prev" title="numpy.testing.suppress_warnings.record" href="generated/numpy.testing.suppress_warnings.record.html" > 
  </head>
  <body>
<div class="container">
  <div class="top-scipy-org-logo-header" style="background-color: #a2bae8;">
    <a href="../index.html">
      <img border=0 alt="NumPy" src="../_static/numpy_logo.png"></a>
    </div>
  </div>
</div>


    <div class="container">
      <div class="main">
        
	<div class="row-fluid">
	  <div class="span12">
	    <div class="spc-navbar">
              
    <ul class="nav nav-pills pull-left">
        <li class="active"><a href="https://numpy.org/">NumPy.org</a></li>
        <li class="active"><a href="https://numpy.org/doc">Docs</a></li>
        
        <li class="active"><a href="../index.html">NumPy v1.18 Manual</a></li>
        

          <li class="active"><a href="index.html" >NumPy Reference</a></li>
          <li class="active"><a href="routines.html" >Routines</a></li>
          <li class="active"><a href="routines.testing.html" accesskey="U">Test Support (<code class="xref py py-mod docutils literal notranslate"><span class="pre">numpy.testing</span></code>)</a></li> 
    </ul>
              
              
    <ul class="nav nav-pills pull-right">
      <li class="active">
        <a href="../genindex.html" title="General Index"
           accesskey="I">index</a>
      </li>
      <li class="active">
        <a href="routines.window.html" title="Window functions"
           accesskey="N">next</a>
      </li>
      <li class="active">
        <a href="generated/numpy.testing.suppress_warnings.record.html" title="numpy.testing.suppress_warnings.record"
           accesskey="P">previous</a>
      </li>
    </ul>
              
	    </div>
	  </div>
	</div>
        

	<div class="row-fluid">
      <div class="spc-rightsidebar span3">
        <div class="sphinxsidebarwrapper">
  <h3><a href="../contents.html">Table of Contents</a></h3>
  <ul>
<li><a class="reference internal" href="#">Testing Guidelines</a><ul>
<li><a class="reference internal" href="#introduction">Introduction</a></li>
<li><a class="reference internal" href="#writing-your-own-tests">Writing your own tests</a><ul>
<li><a class="reference internal" href="#labeling-tests">Labeling tests</a></li>
<li><a class="reference internal" href="#easier-setup-and-teardown-functions-methods">Easier setup and teardown functions / methods</a></li>
<li><a class="reference internal" href="#parametric-tests">Parametric tests</a></li>
<li><a class="reference internal" href="#doctests">Doctests</a></li>
<li><a class="reference internal" href="#tests"><code class="docutils literal notranslate"><span class="pre">tests/</span></code></a></li>
<li><a class="reference internal" href="#init-py-and-setup-py"><code class="docutils literal notranslate"><span class="pre">__init__.py</span></code> and <code class="docutils literal notranslate"><span class="pre">setup.py</span></code></a></li>
</ul>
</li>
<li><a class="reference internal" href="#tips-tricks">Tips &amp; Tricks</a><ul>
<li><a class="reference internal" href="#creating-many-similar-tests">Creating many similar tests</a></li>
<li><a class="reference internal" href="#known-failures-skipping-tests">Known failures &amp; skipping tests</a></li>
<li><a class="reference internal" href="#tests-on-random-data">Tests on random data</a></li>
</ul>
</li>
</ul>
</li>
</ul>

  <h4>Previous topic</h4>
  <p class="topless"><a href="generated/numpy.testing.suppress_warnings.record.html"
                        title="previous chapter">numpy.testing.suppress_warnings.record</a></p>
  <h4>Next topic</h4>
  <p class="topless"><a href="routines.window.html"
                        title="next chapter">Window functions</a></p>
<div id="searchbox" style="display: none" role="search">
  <h4>Quick search</h4>
    <div>
    <form class="search" action="../search.html" method="get">
      <input type="text" style="width: inherit;" name="q" />
      <input type="submit" value="search" />
      <input type="hidden" name="check_keywords" value="yes" />
      <input type="hidden" name="area" value="default" />
    </form>
    </div>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
        </div>
      </div>
          <div class="span9">
            
        <div class="bodywrapper">
          <div class="body" id="spc-section-body">
            
  <div class="section" id="testing-guidelines">
<span id="id1"></span><h1>Testing Guidelines<a class="headerlink" href="#testing-guidelines" title="Permalink to this headline">¶</a></h1>
<div class="section" id="introduction">
<h2>Introduction<a class="headerlink" href="#introduction" title="Permalink to this headline">¶</a></h2>
<p>Until the 1.15 release, NumPy used the <a class="reference external" href="https://nose.readthedocs.io/en/latest/">nose</a> testing framework, it now uses
the <a class="reference external" href="https://pytest.readthedocs.io">pytest</a> framework. The older framework is still maintained in order to
support downstream projects that use the old numpy framework, but all tests
for NumPy should use pytest.</p>
<p>Our goal is that every module and package in SciPy and NumPy
should have a thorough set of unit
tests. These tests should exercise the full functionality of a given
routine as well as its robustness to erroneous or unexpected input
arguments. Long experience has shown that by far the best time to
write the tests is before you write or change the code - this is
<a class="reference external" href="https://en.wikipedia.org/wiki/Test-driven_development">test-driven development</a>.  The
arguments for this can sound rather abstract, but we can assure you
that you will find that writing the tests first leads to more robust
and better designed code. Well-designed tests with good coverage make
an enormous difference to the ease of refactoring. Whenever a new bug
is found in a routine, you should write a new test for that specific
case and add it to the test suite to prevent that bug from creeping
back in unnoticed.</p>
<p>To run SciPy’s full test suite, use the following:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="kn">import</span> <span class="nn">scipy</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">scipy</span><span class="o">.</span><span class="n">test</span><span class="p">()</span>
</pre></div>
</div>
<p>or from the command line:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ python runtests.py
</pre></div>
</div>
<p>SciPy uses the testing framework from <a class="reference internal" href="routines.testing.html#module-numpy.testing" title="numpy.testing"><code class="xref py py-mod docutils literal notranslate"><span class="pre">numpy.testing</span></code></a>, so all
the SciPy examples shown here are also applicable to NumPy.  NumPy’s full test
suite can be run as follows:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="kn">import</span> <span class="nn">numpy</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">numpy</span><span class="o">.</span><span class="n">test</span><span class="p">()</span>
</pre></div>
</div>
<p>The test method may take two or more arguments; the first, <code class="docutils literal notranslate"><span class="pre">label</span></code> is a
string specifying what should be tested and the second, <code class="docutils literal notranslate"><span class="pre">verbose</span></code> is an
integer giving the level of output verbosity. See the docstring for
numpy.test for details.  The default value for <code class="docutils literal notranslate"><span class="pre">label</span></code> is ‘fast’ - which
will run the standard tests.  The string ‘full’ will run the full battery
of tests, including those identified as being slow to run. If <code class="docutils literal notranslate"><span class="pre">verbose</span></code>
is 1 or less, the tests will just show information messages about the tests
that are run; but if it is greater than 1, then the tests will also provide
warnings on missing tests. So if you want to run every test and get
messages about which modules don’t have tests:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">scipy</span><span class="o">.</span><span class="n">test</span><span class="p">(</span><span class="n">label</span><span class="o">=</span><span class="s1">&#39;full&#39;</span><span class="p">,</span> <span class="n">verbose</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span> <span class="c1"># or scipy.test(&#39;full&#39;, 2)</span>
</pre></div>
</div>
<p>Finally, if you are only interested in testing a subset of SciPy, for
example, the <code class="docutils literal notranslate"><span class="pre">integrate</span></code> module, use the following:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">scipy</span><span class="o">.</span><span class="n">integrate</span><span class="o">.</span><span class="n">test</span><span class="p">()</span>
</pre></div>
</div>
<p>or from the command line:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$python runtests.py -t scipy/integrate/tests
</pre></div>
</div>
<p>The rest of this page will give you a basic idea of how to add unit
tests to modules in SciPy. It is extremely important for us to have
extensive unit testing since this code is going to be used by
scientists and researchers and is being developed by a large number of
people spread across the world. So, if you are writing a package that
you’d like to become part of SciPy, please write the tests as you
develop the package. Also since much of SciPy is legacy code that was
originally written without unit tests, there are still several modules
that don’t have tests yet. Please feel free to choose one of these
modules and develop tests for it as you read through
this introduction.</p>
</div>
<div class="section" id="writing-your-own-tests">
<h2>Writing your own tests<a class="headerlink" href="#writing-your-own-tests" title="Permalink to this headline">¶</a></h2>
<p>Every Python module, extension module, or subpackage in the SciPy
package directory should have a corresponding <code class="docutils literal notranslate"><span class="pre">test_&lt;name&gt;.py</span></code> file.
Pytest examines these files for test methods (named test*) and test
classes (named Test*).</p>
<p>Suppose you have a SciPy module <code class="docutils literal notranslate"><span class="pre">scipy/xxx/yyy.py</span></code> containing a
function <code class="docutils literal notranslate"><span class="pre">zzz()</span></code>.  To test this function you would create a test
module called <code class="docutils literal notranslate"><span class="pre">test_yyy.py</span></code>.  If you only need to test one aspect of
<code class="docutils literal notranslate"><span class="pre">zzz</span></code>, you can simply add a test function:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">test_zzz</span><span class="p">():</span>
    <span class="n">assert_</span><span class="p">(</span><span class="n">zzz</span><span class="p">()</span> <span class="o">==</span> <span class="s1">&#39;Hello from zzz&#39;</span><span class="p">)</span>
</pre></div>
</div>
<p>More often, we need to group a number of tests together, so we create
a test class:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">numpy.testing</span> <span class="kn">import</span> <span class="n">assert_</span><span class="p">,</span> <span class="n">assert_raises</span>

<span class="c1"># import xxx symbols</span>
<span class="kn">from</span> <span class="nn">scipy.xxx.yyy</span> <span class="kn">import</span> <span class="n">zzz</span>

<span class="k">class</span> <span class="nc">TestZzz</span><span class="p">:</span>
    <span class="k">def</span> <span class="nf">test_simple</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="n">assert_</span><span class="p">(</span><span class="n">zzz</span><span class="p">()</span> <span class="o">==</span> <span class="s1">&#39;Hello from zzz&#39;</span><span class="p">)</span>

    <span class="k">def</span> <span class="nf">test_invalid_parameter</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="n">assert_raises</span><span class="p">(</span><span class="o">...</span><span class="p">)</span>
</pre></div>
</div>
<p>Within these test methods, <code class="docutils literal notranslate"><span class="pre">assert_()</span></code> and related functions are used to test
whether a certain assumption is valid. If the assertion fails, the test fails.
Note that the Python builtin <code class="docutils literal notranslate"><span class="pre">assert</span></code> should not be used, because it is
stripped during compilation with <code class="docutils literal notranslate"><span class="pre">-O</span></code>.</p>
<p>Note that <code class="docutils literal notranslate"><span class="pre">test_</span></code> functions or methods should not have a docstring, because
that makes it hard to identify the test from the output of running the test
suite with <code class="docutils literal notranslate"><span class="pre">verbose=2</span></code> (or similar verbosity setting).  Use plain comments
(<code class="docutils literal notranslate"><span class="pre">#</span></code>) if necessary.</p>
<div class="section" id="labeling-tests">
<h3>Labeling tests<a class="headerlink" href="#labeling-tests" title="Permalink to this headline">¶</a></h3>
<p>As an alternative to <code class="docutils literal notranslate"><span class="pre">pytest.mark.&lt;label&gt;</span></code>, there are a number of labels you
can use.</p>
<p>Unlabeled tests like the ones above are run in the default
<code class="docutils literal notranslate"><span class="pre">scipy.test()</span></code> run.  If you want to label your test as slow - and
therefore reserved for a full <code class="docutils literal notranslate"><span class="pre">scipy.test(label='full')</span></code> run, you
can label it with a decorator:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="c1"># numpy.testing module includes &#39;import decorators as dec&#39;</span>
<span class="kn">from</span> <span class="nn">numpy.testing</span> <span class="kn">import</span> <span class="n">dec</span><span class="p">,</span> <span class="n">assert_</span>

<span class="nd">@dec</span><span class="o">.</span><span class="n">slow</span>
<span class="k">def</span> <span class="nf">test_big</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
    <span class="nb">print</span> <span class="s1">&#39;Big, slow test&#39;</span>
</pre></div>
</div>
<p>Similarly for methods:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">test_zzz</span><span class="p">:</span>
    <span class="nd">@dec</span><span class="o">.</span><span class="n">slow</span>
    <span class="k">def</span> <span class="nf">test_simple</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="n">assert_</span><span class="p">(</span><span class="n">zzz</span><span class="p">()</span> <span class="o">==</span> <span class="s1">&#39;Hello from zzz&#39;</span><span class="p">)</span>
</pre></div>
</div>
<p>Available labels are:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">slow</span></code>: marks a test as taking a long time</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">setastest(tf)</span></code>: work-around for test discovery when the test name is
non conformant</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">skipif(condition,</span> <span class="pre">msg=None)</span></code>: skips the test when <code class="docutils literal notranslate"><span class="pre">eval(condition)</span></code> is
<code class="docutils literal notranslate"><span class="pre">True</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">knownfailureif(fail_cond,</span> <span class="pre">msg=None)</span></code>: will avoid running the test if
<code class="docutils literal notranslate"><span class="pre">eval(fail_cond)</span></code> is <code class="docutils literal notranslate"><span class="pre">True</span></code>, useful for tests that conditionally segfault</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">deprecated(conditional=True)</span></code>: filters deprecation warnings emitted in the
test</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">paramaterize(var,</span> <span class="pre">input)</span></code>: an alternative to
<a class="reference external" href="https://docs.pytest.org/en/latest/parametrize.html">pytest.mark.paramaterized</a></p></li>
</ul>
</div>
<div class="section" id="easier-setup-and-teardown-functions-methods">
<h3>Easier setup and teardown functions / methods<a class="headerlink" href="#easier-setup-and-teardown-functions-methods" title="Permalink to this headline">¶</a></h3>
<p>Testing looks for module-level or class-level setup and teardown functions by
name; thus:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">setup</span><span class="p">():</span>
    <span class="sd">&quot;&quot;&quot;Module-level setup&quot;&quot;&quot;</span>
    <span class="nb">print</span> <span class="s1">&#39;doing setup&#39;</span>

<span class="k">def</span> <span class="nf">teardown</span><span class="p">():</span>
    <span class="sd">&quot;&quot;&quot;Module-level teardown&quot;&quot;&quot;</span>
    <span class="nb">print</span> <span class="s1">&#39;doing teardown&#39;</span>


<span class="k">class</span> <span class="nc">TestMe</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
    <span class="k">def</span> <span class="nf">setup</span><span class="p">():</span>
        <span class="sd">&quot;&quot;&quot;Class-level setup&quot;&quot;&quot;</span>
        <span class="nb">print</span> <span class="s1">&#39;doing setup&#39;</span>

    <span class="k">def</span> <span class="nf">teardown</span><span class="p">():</span>
        <span class="sd">&quot;&quot;&quot;Class-level teardown&quot;&quot;&quot;</span>
        <span class="nb">print</span> <span class="s1">&#39;doing teardown&#39;</span>
</pre></div>
</div>
<p>Setup and teardown functions to functions and methods are known as “fixtures”,
and their use is not encouraged.</p>
</div>
<div class="section" id="parametric-tests">
<h3>Parametric tests<a class="headerlink" href="#parametric-tests" title="Permalink to this headline">¶</a></h3>
<p>One very nice feature of testing is allowing easy testing across a range
of parameters - a nasty problem for standard unit tests. Use the
<code class="docutils literal notranslate"><span class="pre">dec.paramaterize</span></code> decorator.</p>
</div>
<div class="section" id="doctests">
<h3>Doctests<a class="headerlink" href="#doctests" title="Permalink to this headline">¶</a></h3>
<p>Doctests are a convenient way of documenting the behavior of a function
and allowing that behavior to be tested at the same time.  The output
of an interactive Python session can be included in the docstring of a
function, and the test framework can run the example and compare the
actual output to the expected output.</p>
<p>The doctests can be run by adding the <code class="docutils literal notranslate"><span class="pre">doctests</span></code> argument to the
<code class="docutils literal notranslate"><span class="pre">test()</span></code> call; for example, to run all tests (including doctests)
for numpy.lib:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">np</span><span class="o">.</span><span class="n">lib</span><span class="o">.</span><span class="n">test</span><span class="p">(</span><span class="n">doctests</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</pre></div>
</div>
<p>The doctests are run as if they are in a fresh Python instance which
has executed <code class="docutils literal notranslate"><span class="pre">import</span> <span class="pre">numpy</span> <span class="pre">as</span> <span class="pre">np</span></code>. Tests that are part of a SciPy
subpackage will have that subpackage already imported. E.g. for a test
in <code class="docutils literal notranslate"><span class="pre">scipy/linalg/tests/</span></code>, the namespace will be created such that
<code class="docutils literal notranslate"><span class="pre">from</span> <span class="pre">scipy</span> <span class="pre">import</span> <span class="pre">linalg</span></code> has already executed.</p>
</div>
<div class="section" id="tests">
<h3><code class="docutils literal notranslate"><span class="pre">tests/</span></code><a class="headerlink" href="#tests" title="Permalink to this headline">¶</a></h3>
<p>Rather than keeping the code and the tests in the same directory, we
put all the tests for a given subpackage in a <code class="docutils literal notranslate"><span class="pre">tests/</span></code>
subdirectory. For our example, if it doesn’t already exist you will
need to create a <code class="docutils literal notranslate"><span class="pre">tests/</span></code> directory in <code class="docutils literal notranslate"><span class="pre">scipy/xxx/</span></code>. So the path
for <code class="docutils literal notranslate"><span class="pre">test_yyy.py</span></code> is <code class="docutils literal notranslate"><span class="pre">scipy/xxx/tests/test_yyy.py</span></code>.</p>
<p>Once the <code class="docutils literal notranslate"><span class="pre">scipy/xxx/tests/test_yyy.py</span></code> is written, its possible to
run the tests by going to the <code class="docutils literal notranslate"><span class="pre">tests/</span></code> directory and typing:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">python</span> <span class="n">test_yyy</span><span class="o">.</span><span class="n">py</span>
</pre></div>
</div>
<p>Or if you add <code class="docutils literal notranslate"><span class="pre">scipy/xxx/tests/</span></code> to the Python path, you could run
the tests interactively in the interpreter like this:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="kn">import</span> <span class="nn">test_yyy</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">test_yyy</span><span class="o">.</span><span class="n">test</span><span class="p">()</span>
</pre></div>
</div>
</div>
<div class="section" id="init-py-and-setup-py">
<h3><code class="docutils literal notranslate"><span class="pre">__init__.py</span></code> and <code class="docutils literal notranslate"><span class="pre">setup.py</span></code><a class="headerlink" href="#init-py-and-setup-py" title="Permalink to this headline">¶</a></h3>
<p>Usually, however, adding the <code class="docutils literal notranslate"><span class="pre">tests/</span></code> directory to the python path
isn’t desirable. Instead it would better to invoke the test straight
from the module <code class="docutils literal notranslate"><span class="pre">xxx</span></code>. To this end, simply place the following lines
at the end of your package’s <code class="docutils literal notranslate"><span class="pre">__init__.py</span></code> file:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="o">...</span>
<span class="k">def</span> <span class="nf">test</span><span class="p">(</span><span class="n">level</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">verbosity</span><span class="o">=</span><span class="mi">1</span><span class="p">):</span>
    <span class="kn">from</span> <span class="nn">numpy.testing</span> <span class="kn">import</span> <span class="n">Tester</span>
    <span class="k">return</span> <span class="n">Tester</span><span class="p">()</span><span class="o">.</span><span class="n">test</span><span class="p">(</span><span class="n">level</span><span class="p">,</span> <span class="n">verbosity</span><span class="p">)</span>
</pre></div>
</div>
<p>You will also need to add the tests directory in the configuration
section of your setup.py:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="o">...</span>
<span class="k">def</span> <span class="nf">configuration</span><span class="p">(</span><span class="n">parent_package</span><span class="o">=</span><span class="s1">&#39;&#39;</span><span class="p">,</span> <span class="n">top_path</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
    <span class="o">...</span>
    <span class="n">config</span><span class="o">.</span><span class="n">add_data_dir</span><span class="p">(</span><span class="s1">&#39;tests&#39;</span><span class="p">)</span>
    <span class="k">return</span> <span class="n">config</span>
<span class="o">...</span>
</pre></div>
</div>
<p>Now you can do the following to test your module:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="kn">import</span> <span class="nn">scipy</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">scipy</span><span class="o">.</span><span class="n">xxx</span><span class="o">.</span><span class="n">test</span><span class="p">()</span>
</pre></div>
</div>
<p>Also, when invoking the entire SciPy test suite, your tests will be
found and run:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="kn">import</span> <span class="nn">scipy</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">scipy</span><span class="o">.</span><span class="n">test</span><span class="p">()</span>
<span class="go"># your tests are included and run automatically!</span>
</pre></div>
</div>
</div>
</div>
<div class="section" id="tips-tricks">
<h2>Tips &amp; Tricks<a class="headerlink" href="#tips-tricks" title="Permalink to this headline">¶</a></h2>
<div class="section" id="creating-many-similar-tests">
<h3>Creating many similar tests<a class="headerlink" href="#creating-many-similar-tests" title="Permalink to this headline">¶</a></h3>
<p>If you have a collection of tests that must be run multiple times with
minor variations, it can be helpful to create a base class containing
all the common tests, and then create a subclass for each variation.
Several examples of this technique exist in NumPy; below are excerpts
from one in <a class="reference external" href="https://github.com/numpy/numpy/blob/master/numpy/linalg/tests/test_linalg.py">numpy/linalg/tests/test_linalg.py</a>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">LinalgTestCase</span><span class="p">:</span>
    <span class="k">def</span> <span class="nf">test_single</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="n">a</span> <span class="o">=</span> <span class="n">array</span><span class="p">([[</span><span class="mf">1.</span><span class="p">,</span><span class="mf">2.</span><span class="p">],</span> <span class="p">[</span><span class="mf">3.</span><span class="p">,</span><span class="mf">4.</span><span class="p">]],</span> <span class="n">dtype</span><span class="o">=</span><span class="n">single</span><span class="p">)</span>
        <span class="n">b</span> <span class="o">=</span> <span class="n">array</span><span class="p">([</span><span class="mf">2.</span><span class="p">,</span> <span class="mf">1.</span><span class="p">],</span> <span class="n">dtype</span><span class="o">=</span><span class="n">single</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">do</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">)</span>

    <span class="k">def</span> <span class="nf">test_double</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="n">a</span> <span class="o">=</span> <span class="n">array</span><span class="p">([[</span><span class="mf">1.</span><span class="p">,</span><span class="mf">2.</span><span class="p">],</span> <span class="p">[</span><span class="mf">3.</span><span class="p">,</span><span class="mf">4.</span><span class="p">]],</span> <span class="n">dtype</span><span class="o">=</span><span class="n">double</span><span class="p">)</span>
        <span class="n">b</span> <span class="o">=</span> <span class="n">array</span><span class="p">([</span><span class="mf">2.</span><span class="p">,</span> <span class="mf">1.</span><span class="p">],</span> <span class="n">dtype</span><span class="o">=</span><span class="n">double</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">do</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">)</span>

    <span class="o">...</span>

<span class="k">class</span> <span class="nc">TestSolve</span><span class="p">(</span><span class="n">LinalgTestCase</span><span class="p">):</span>
    <span class="k">def</span> <span class="nf">do</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">):</span>
        <span class="n">x</span> <span class="o">=</span> <span class="n">linalg</span><span class="o">.</span><span class="n">solve</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">)</span>
        <span class="n">assert_almost_equal</span><span class="p">(</span><span class="n">b</span><span class="p">,</span> <span class="n">dot</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">x</span><span class="p">))</span>
        <span class="n">assert_</span><span class="p">(</span><span class="n">imply</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">b</span><span class="p">,</span> <span class="n">matrix</span><span class="p">),</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">matrix</span><span class="p">)))</span>

<span class="k">class</span> <span class="nc">TestInv</span><span class="p">(</span><span class="n">LinalgTestCase</span><span class="p">):</span>
    <span class="k">def</span> <span class="nf">do</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">):</span>
        <span class="n">a_inv</span> <span class="o">=</span> <span class="n">linalg</span><span class="o">.</span><span class="n">inv</span><span class="p">(</span><span class="n">a</span><span class="p">)</span>
        <span class="n">assert_almost_equal</span><span class="p">(</span><span class="n">dot</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">a_inv</span><span class="p">),</span> <span class="n">identity</span><span class="p">(</span><span class="n">asarray</span><span class="p">(</span><span class="n">a</span><span class="p">)</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]))</span>
        <span class="n">assert_</span><span class="p">(</span><span class="n">imply</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">matrix</span><span class="p">),</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">a_inv</span><span class="p">,</span> <span class="n">matrix</span><span class="p">)))</span>
</pre></div>
</div>
<p>In this case, we wanted to test solving a linear algebra problem using
matrices of several data types, using <code class="docutils literal notranslate"><span class="pre">linalg.solve</span></code> and
<code class="docutils literal notranslate"><span class="pre">linalg.inv</span></code>.  The common test cases (for single-precision,
double-precision, etc. matrices) are collected in <code class="docutils literal notranslate"><span class="pre">LinalgTestCase</span></code>.</p>
</div>
<div class="section" id="known-failures-skipping-tests">
<h3>Known failures &amp; skipping tests<a class="headerlink" href="#known-failures-skipping-tests" title="Permalink to this headline">¶</a></h3>
<p>Sometimes you might want to skip a test or mark it as a known failure,
such as when the test suite is being written before the code it’s
meant to test, or if a test only fails on a particular architecture.</p>
<p>To skip a test, simply use <code class="docutils literal notranslate"><span class="pre">skipif</span></code>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">pytest</span>

<span class="nd">@pytest</span><span class="o">.</span><span class="n">mark</span><span class="o">.</span><span class="n">skipif</span><span class="p">(</span><span class="n">SkipMyTest</span><span class="p">,</span> <span class="n">reason</span><span class="o">=</span><span class="s2">&quot;Skipping this test because...&quot;</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_something</span><span class="p">(</span><span class="n">foo</span><span class="p">):</span>
    <span class="o">...</span>
</pre></div>
</div>
<p>The test is marked as skipped if <code class="docutils literal notranslate"><span class="pre">SkipMyTest</span></code> evaluates to nonzero,
and the message in verbose test output is the second argument given to
<code class="docutils literal notranslate"><span class="pre">skipif</span></code>.  Similarly, a test can be marked as a known failure by
using <code class="docutils literal notranslate"><span class="pre">xfail</span></code>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">pytest</span>

<span class="nd">@pytest</span><span class="o">.</span><span class="n">mark</span><span class="o">.</span><span class="n">xfail</span><span class="p">(</span><span class="n">MyTestFails</span><span class="p">,</span> <span class="n">reason</span><span class="o">=</span><span class="s2">&quot;This test is known to fail because...&quot;</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_something_else</span><span class="p">(</span><span class="n">foo</span><span class="p">):</span>
    <span class="o">...</span>
</pre></div>
</div>
<p>Of course, a test can be unconditionally skipped or marked as a known
failure by using <code class="docutils literal notranslate"><span class="pre">skip</span></code> or <code class="docutils literal notranslate"><span class="pre">xfail</span></code> without argument, respectively.</p>
<p>A total of the number of skipped and known failing tests is displayed
at the end of the test run.  Skipped tests are marked as <code class="docutils literal notranslate"><span class="pre">'S'</span></code> in
the test results (or <code class="docutils literal notranslate"><span class="pre">'SKIPPED'</span></code> for <code class="docutils literal notranslate"><span class="pre">verbose</span> <span class="pre">&gt;</span> <span class="pre">1</span></code>), and known
failing tests are marked as <code class="docutils literal notranslate"><span class="pre">'x'</span></code> (or <code class="docutils literal notranslate"><span class="pre">'XFAIL'</span></code> if <code class="docutils literal notranslate"><span class="pre">verbose</span> <span class="pre">&gt;</span>
<span class="pre">1</span></code>).</p>
</div>
<div class="section" id="tests-on-random-data">
<h3>Tests on random data<a class="headerlink" href="#tests-on-random-data" title="Permalink to this headline">¶</a></h3>
<p>Tests on random data are good, but since test failures are meant to expose
new bugs or regressions, a test that passes most of the time but fails
occasionally with no code changes is not helpful. Make the random data
deterministic by setting the random number seed before generating it.  Use
either Python’s <code class="docutils literal notranslate"><span class="pre">random.seed(some_number)</span></code> or NumPy’s
<code class="docutils literal notranslate"><span class="pre">numpy.random.seed(some_number)</span></code>, depending on the source of random numbers.</p>
</div>
</div>
</div>


          </div>
        </div>
          </div>
        </div>
      </div>
    </div>

    <div class="container container-navbar-bottom">
      <div class="spc-navbar">
        
      </div>
    </div>
    <div class="container">
    <div class="footer">
    <div class="row-fluid">
    <ul class="inline pull-left">
      <li>
        &copy; Copyright 2008-2019, The SciPy community.
      </li>
      <li>
      Last updated on Feb 20, 2020.
      </li>
      <li>
      Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> 2.4.2.
      </li>
    </ul>
    </div>
    </div>
    </div>
  </body>
</html>